An opinionated minimal-compromises guide to configuring a maximally secure server for high stakes use cases where privacy and security are favored over compatibility, cost, or effeciency.
This intends to be largely a showcase of the work of others and act as a starting point for researching this space.
- Target protects:
- Automated air/ground transportation
- Nuclear weapons
- Electric grid
- Medical implants
- Dive computers
- Secrets that could end any entity
- Access to unlimited financial gain
- Human lives
- Attacker has
- No ethics
- unlimited funding
- decades of patience
- Knowledge deeper than yours of every component
- 0-days of any currently known class
- Attacker can
- compromise any single point in the supply chain
- compromise any single system
- compromise any single individual
- Attacker wants
- Theft (cryptocurrency, bank accounts, stock tips, blackmail, databases)
- Sabotage (to a company or country for any reason)
- Chaos (May not be rational)
- Favor security and privacy over efficiency
- Every system:
- is treated as a single purpose immutable appliance
- replaced not updated
- Every component must be:
- auditable by anyone
- reproducible deterministically by anyone
- audited by multiple reputable third parties.
- fail on any unathorized physical tampering attempt
- handle cryptographic operations in constant time
- maintain secret keys physically separate from networks
- have the bare minimum resources to complete its intended function
- Decentralized
- Multisig
- Coding practices
- review practices
- Signed reproducible builds must be possible
- Code must be signed with a well-known key of author and ideally reviewer(s)
- Consider reviews by any distribution channel maintainers
- Always assume force pushes and tag clobbers: pin hashes
- Assume upstreams will vanish without warning: mirror everything yourself
- Reproducible builds
- Signing
Static analysis analyzes source code or compiled binaries for security flaws. A critical part of a security focused software development is using tools to help catch common human errors before code hits production.
Memory bugs are most common and where possible one should favor memory safe languages designed for security: Rust, Go, OCaml, Zig
- Google CodeSearchDiggity
- Graudit
- LGTM
- SonarQube
- VisualCodeGrepper
- shellcheck
- synode
- BOON
- CQual
- xg++
- Eau Claire tool
- MOPS
- Split
- Flawfinder
- PreFast
- Puma Scan
- .Net Security Guard
- Bandit
- Brakeman
- Codesake Dawn
- SpotBugs
- PMD
- progpilot
- RIPS
- pgpcs-security-audit
Everything on unix is a file, and as such filesystem mount options and permissions are one of the most effective ways to restrict what can or can't be done in a given directory.
Everything should be either a read-only filesystem like quashfs or a tmpfs. Never allow writes to root filesystem.
This is all managed via /etc/fstab
- Usage:
proc /proc proc defaults,hidepid=2 0 0
- Usage:
udev /dev devtmpfs defaults,nosuid,noexec,noatime 0 0
- Usage:
devpts /dev/pts devpts defaults,newinstance,ptmxmode=0666 0 0
- Usage:
tmpfs /dev/shm tmpfs defaults,nodev,nosuid,noexec 0 0
- Usage:
tmpfs /tmp tmpfs nodev,nosuid,noexec,size=2G 0 0
- Usage:
/tmp /var/tmp none rw,noexec,nosuid,nodev,bind 0 0
- luks
- luks + sgx
- Cold boot attacks
- Usage:
-mindirect-branch=thunk-extern - Intention:
- Speculative execution is incompatible with call/return thunks
- Convert indirect calls and jumps to call and return thunks
- Used in Spectre v2 attack mitigations to prevent CPU branch speculation
- Notes:
- thunk: create one thunk section per input file
- thunk-inline: create one per indirect branch or function return
- thunk-extern: one thunk section for entire program in separate object file
- Resources:
- Usage:
-fstack-protector-strong - Intention:
- Plant random "canary" integers just before stack return pointers
- Buffer overflows hijacking return pointer will normally modify canary
- Ensure canary is still present before a routine uses a pointer on stack
- Resources:
- Usage:
- Statically compiled:
-static-pie - Allow linking PICs:
-fPIE -pie
- Statically compiled:
- Intention:
- Only allow PIC libraries to be linked into executables
- When used in with ASLR, all program memory is allocated randomly together
- Increase difficulty of using exploits that assume a specific memory layout
- Resources:
- Usage:
-fpic -shared - Intention:
- Use in conjunction with PIE
- Compile shared libraries without text relocations
- Allow shared libraries to be safely linked into a PIE
- Resources:
- Usage:
-fstack-clash-protection - Intention:
- Mitigate attacks that rely on colliding neighboring memory regions
- Defeats most historical stack clashing exploits
- Resources:
- Usage:
-Wl,-z,noexecstack -Wl,-z,noexecheap - Intention:
- Buffer overflows tend to put code in programs stack and jump to it
- If all writable addresses are non-executable, the attack is prevented
- Don't mark memory as executable when it is not required
- ELF headers are marked with PT_GNU_STACK and PT_GNU_HEAP
- Set stacks/heaps to be executable only if segment flag calls for it
- Resources:
- Usage:
-DFORTIFY_SOURCE=2 - Intention:
- Many programs rely on functions that are not aware of buffer-length
- Buffer overflow exploits often take advantage of these functions
- Examples include strncpy, strcpy, memcpy, memset
- Fail compilation if these are used in obviously unsafe way.
- Compile with buffer-length aware checks for added run-time detection
- Kill execution if buffer overflow check fires
- Resources:
- Usage:
-Wp, -D_GLIBCXX_ASSERTIONS - Intention:
- Turn on cheap range checks for C++ arrays, vectors, and strings
- Add Null pointer checks when dereferencing smart pointers
- Resources:
- Usage:
-plugin=annobin - Intention:
- Include extra metadata in binary files to assist static analysis tools
- Can be used by scripts to verify hardening options or ABI conflicts
- Resources:
- Usage:
-mzero-caller-saved-regs=all - Notes:
- Requires GCC patch
- Intention:
- Clear caller-saved general registers on function return
- Make ROP, COP, and JOP attacks harder
- Resources:
- Usage:
-mcet -fcf-protection=full - Notes:
- Can only be used -future- Intel CPUs
- Intention:
- Leverage a read-only "shadow stack" preventing injection of special entries
- Check for valid target addresses of control-flow transfers
- Prevent diverting flow of control to an unexpected target
- Intel claims this will replace retpolines to stop Spectre v2 attacks
- Resources:
- Usage:
-Werror=format-security - Intention:
- Error on poorly defined format functions that can be exploited
- Expect string literal and format arguments for sprintf/scanf and similar
- Resources:
- Usage:
-Werror=implicit-function-declaration - Intention:
- Undeclared functions can be interpreted by compilers in unexpected ways
- Ensure undeclared functions break builds so they can be addressed
- Intention:
- Usage:
-Wl,-z,defs - Intention:
- Many attacks rely on taking control of undefined symbols
- Disallow undefined symbols when creating object files
- Resources:
- Usage:
-Wl,-z,relro,-z,now - Intention:
- Lazy binding loads libraries into memory when first accessed
- A reference is left in the Global Offset Table in a known location
- Having a well known memory addreseses makes an attackers job easier
- Make shared libraries read-only after dynamic relocations are applied
- Limit ability of attacker to overwrite GOT entries to shared functions
- Disable lazy binding and force "now" binding
- Note:
- Will result in slower application startup
- Resources:
- Notes:
- Small binaries (~13k hello world vs ~600k with glibc)
- Small .a/.so footprint (~1MB vs ~10MB for glibc)
- Small codebase designed to be easy to audit
- Supported by security focused Linux distros (linuxkit, gentoo, alpine)
- Low-memory or resource exhaustion conditions are never fatal
- Wider support for microcontrollers and embedded targets than glibc
- Rapid termination on security violation to limit attacks on error codepaths
- Can build itself with ASLR to catching internal stack-smashing
- Double-Free protection (as possible)
- Moderate heap-overflow detection
- Use PIE together with static-linking
- Limited machine-specific code minimizing chances of minority arch breakage
- Limit buggy translations via forced literal strings without format strings
- Block all LD_* for suid/sgid binaries limiting runtime behavior overrides
- Non-use of arbitrary-size VLA/alloca, minimal dynamic allocation
- Avoid subtle race condition and async-signal safety issues found in glibc
- Attempts to remove all undefined behavior. Less code == less bugs
- Safe, fully-standards-conforming UTF-8 (source of many security bugs)
- Consistent results even under transient errors without silent fallbacks
- Lazy/Late allocations that would abort on failure are unsupported
- Resources:
- http://www.etalabs.net/compare_libcs.html
- https://gcc.gnu.org/onlinedocs/gcc/Code-Gen-Options.html
- https://www.owasp.org/index.php/C-Based_Toolchain_Hardening#GCC.2FBinutils
- http://www.trapkit.de/tools/checksec.html
While Linux is certianly not designed for out-of-the-box high security it is the most portable for the widest range of use cases and has the largest number of deployments so advice in this section will assume it.
If your application does not require a Linux kernel it is suggested the reader carefully consider security-focused alternatives like OpenBSD, FreeBSD, FreeRTOS, or seL4.
Some of these features don't ship with any published binary kernels for any major distribution so it is assumed the reader will compile their own kernel with a hardened toolchain following the advice in the Toolchain section of this document.
- Platforms: x86_64, arm64
- Intention:
- Enable sanity checks (F), Redzoning (Z), and Poisoning (P)
- Set slub debugging to poison mode
- Resources:
2:
- Usage:
kernel.kptr_restrict = 1 - Intention:
- Attackers often seek to write to kernel writable structures
- Hide kernel pointers normally present in /proc
- Exploits have harder time discovering kernel addresses/symbols
- Resources:
- Usage:
kernel.dmesg_restrict = 1 - Intention:
- Kernel logs often contain sensitive information like memory addresses
- Forbid dmesg access to binaries that lack CAP_SYS_ADMIN capability.
- Resources:
- Usage:
kernel.perf_event_paranoid = 3 - Intention:
- Access to kernel perf event interface aids adversaries in targets attacks
- Disallow access to CPU events, kernel profiling, tracepoints etc
- Resources:
- Usage:
kernel.kexec_load_disabled = 1 - Intention:
- The Kexec interface is complex, and allows replacement of kernel in memory
- This allows fast "reboots" but is very dangerous if exploited
- Kernel memory should be immutable in a conservative security system
- Disable /dev/kmem and any potential kernel exploits that may use it
- Resources:
- Usage:
kernel.yama.ptrace_scope = 3 - Intention:
- Deny ptrace access to any process
- Prevent leaking sensitive memory to malware via debug interfaces
- Notes:
- Will make debugging tools like strace and gdb unusuable
- Resources:
- Usage:
user.max_user_namespaces = 0 - Intention:
- User namespaces are buggy and have been exploited many times
- Bugs have often resulted in privesc exploits
- Disable feature if you don't need it
- Note:
- Container systems like docker rely heavily on user namespaces
- Resources:
- Usage:
kernel.unprivileged_bpf_disabled = 1 - Intention:
- BPF is a kernel VM that allows unprivileged users to run code in the kernel
- This is added attack surface for most use cases, and should be disabled
- Resources:
- Usage:
net.core.bpf_jit_harden = 2 - Intention:
- Use more secure but slower codepaths for BPF JIT for all users
- Enables blinding and disables some tracing/debugging functionality
- Resources:
- Intention:
- Allow usage of static analysis plugins in GCC at kernel compile time
- Resources:
2: https://www.kernel.org/doc/Documentation/gcc-plugins.txt)
- Platforms: x86_64, arm64
- Intention:
- Exploits often take advantage of uninitalized variables
- Force unconditional initialization of all stack variables
- Resources:
- Intention:
- limit exfiltration of recycled stack memory
- stack poisoning mitigation
- runtime stack overflow detection
- Resources:
- Platforms: x86_64, arm64
- Intention:
- Force all structure initialization before usage by other functions
- Resources:
- Platforms: x86_64, arm64
- Intention:
- Extend CONFIG_GCC_PLUGIN_STRUCTLEAK to include pass-by-reference variables
- Resources:
- Platforms: x86_64, arm64
- Intention:
- Some systems have known issues generating hardware entropy in early boot
- insert local variable in loop counts, cases, branching, etc
- Permuate a global variable based on value changes to marked functions
- Use global variable to help seed early boot entropy pool
- Notes:
- Many of the sources of entropy here are deterministic.
- It is not advised to rely on this plugin alone. Use a TRNG where possible.
- Resources:
- Platforms: x86_64, arm64
- Intention:
- Randomize the layout of selected structures at compile time
- Defend against attacks rely on struct knowledge
- Resources:
- Platforms: x86_64, arm64
- Intention:
- Infinite loops prevent CPUs from speculating the target of an indirect jump
- Attach infinite loop to every return call as a "return trampoline"
- Never actually execute this infinite loop
- Mitigates kernel or cross-process memory disclosure attacks like Spectre
- Notes:
- Use in combination with -mindirect-branch=thunk-extern in GCC8
- Edge case: When an RSB empties, Skylake+ uses vulnerable BTB prediction
- Also apply vendor firmware mitigations where possible
- Resources:
- Retpoline: a software construct for preventing branch-target-injection
- Mitre: CVE-2017-5715
- lkml Retpoline Discission
- Kernel.org: Retpoline patch
- [Intel: Host Firmware Speculative Execution Side Channel Mitigation]
- Platforms: x86_64, arm64
- Intention: *
- Resources:
- Platforms: x86_64, arm64
- Intention: *
- Platforms: x86_64, arm64
- Intention: *
- Platforms: x86_64
- Intention: *
- Platforms: x86_64
- Intention:
- Remove vsyscall entirely avoiding it as a fixed-position ROP target.
- Resources:
2:
- Platforms: x86_64
- Intention: *
- Platforms: x86_64
- Intention:
- Disable 32 bit program emulation and all related attack classes.
- Platforms: x86_64
- Intention: *
- Platforms: x86_64
- Intention: *
- Platforms: arm64
- Intention: *
- Platforms: arm64
- Intention:
- Kernel Page Table Isolation
- Remove an entire class of cache timing side-channels.
- Platforms: x86_64, arm64
- Intention: *
- Platforms: x86_64, arm64
- Intention: *
- Platforms: x86_64, arm64
- Intention: *
- Platforms: x86_64, arm64
- Intention: *
- Platforms: x86_64, arm64
- Intention: *
- Platforms: x86_64, arm64
- Intention: *
- Platforms: x86_64, arm64
- Intention: *
- Platforms: x86_64, arm64
- Intention: *
- Platforms: x86_64, arm64
- Intention: *
- Platforms: x86_64, arm64
- Intention: *
- Platforms: x86_64, arm64
- Intention: *
- Platforms: x86_64, arm64
- Intention: *
- Platforms: x86_64, arm64
- Intention: *
- Resources:
- Platforms: x86_64, arm64
- Intention: *
- Platforms: x86_64, arm64
- Intention: *
- Platforms: x86_64, arm64
- Intention: *
- Platforms: x86_64, arm64
- Intention: *
- Platforms: x86_64, arm64
- Intention: *
- Platforms: x86_64, arm64
- Intention: *
- Platforms: x86_64, arm64
- Intention: *
- Platforms: x86_64, arm64
- Intention: *
- Platforms: x86_64, arm64
- Intention: *
- Platforms: x86_64, arm64
- Intention: *
- Platforms: x86_64, arm64
- Intention: *
- Platforms: x86_64, arm64
- Intention: *
- Platforms: x86_64, arm64
- Intention: *
- Platforms: x86_64, arm64
- Intention: *
- Platforms: x86_64, arm64
- Intention: *
- Platforms: x86_64, arm64
- Intention: *
- Platforms: x86_64, arm64
- Intention: *
- Platforms: x86_64, arm64
- Intention: *
- Platforms: x86_64, arm64
- Intention: *
- Platforms: x86_64, arm64
- Intention: *
- Platforms: x86_64, arm64
- Intention: *
- Platforms: x86_64, arm64
- Intention: *
- Platforms: x86_64, arm64
- Intention: *
- Platforms: x86_64, arm64
- Intention: *
- Platforms: x86_64, arm64
- Intention: *
- Platforms: x86_64, arm64
- Intention: *
- Fedora Hardening Flags
- Android Kernel Hardening
- ChromeOS Kernel Configs
- Debian Hardening
- Ubuntu Compiler Flags
- Arch LInux Security
- Securing Debian Howto
- RedHat: Recommended GCC Compler Flags
- Debian Security Checklist
- System Down - HN discussion
- Why OpenBSD is Important To Me - HN Discussion
- Differences Between ASLR KASLR and KARL
- Linuxkit Security